Image sensor initialization in a robotic vehicle

ABSTRACT

Various embodiments include processing devices and methods for initializing an image sensor of a robotic vehicle. In some embodiments, a processor of the robotic vehicle may extract first features from a captured image of the target image, may extract second features from a predefined image pyramid of the target image, may match first features and second features, and may estimate an image sensor pose based on matched features and known dimensions of the target image. In some embodiments, the processor may estimate an image or to image sensor pose based on matched features and a translation and/or rotation of the robotic vehicle. In some embodiments, the processor may determine a coordinate system for the robotic vehicle based on one or more image sensor poses.

BACKGROUND

Robotic vehicles are being developed for a wide range of applications.Robotic vehicles may be equipped with cameras capable of capturing animage, a sequence of images, or video. Some robotic vehicles may beequipped with a monocular image sensor, such as a monocular camera.

Robotic vehicles may perform simultaneous localization and mapping(SLAM) to construct and update a map of an unknown environment whilesimultaneously keeping track of the robotic vehicle's location withinthe environment. A robotic vehicle may gather data useful for SLAM usingits image sensor device. However, for robotic vehicles equipped with amonocular image sensor, the image sensor may be unable to providesufficient information to enable the robotic vehicle to determine bothimage sensor position and orientation (pose) and map points for theenvironment. Further, for robotic vehicles equipped with a monocularimage sensor, initialization of the image sensor is critical. Errors ininitialization of the monocular image sensor may lead to uncorrectablebias or errors in the robotic vehicle's coordinate system.

SUMMARY

Various embodiments include methods that may be implemented on aprocessor of a robotic vehicle for image sensor initialization of animage sensor of a robotic vehicle.

Various embodiments may include capturing an image of a target imagewith the image sensor of the robotic vehicle, extracting first featuresfrom the captured image, extracting second features from a predefinedimage pyramid of the target image, matching one or more extracted firstfeatures with one or more extracted second features, estimating an imagesensor pose based on matched features and known dimensions of the targetimage, determining whether the image sensor pose is within one or morethresholds, and maneuvering the robotic vehicle to another location fromwhich to capture another image of the target image in response todetermining that the image sensor pose is not within the one or morethresholds.

In some embodiments, the image sensor initialization may include amonocular camera-based simultaneous localization and mapping (SLAM)initialization. Some embodiments may further include determining thatthe image sensor initialization is successful in response to determiningthat the image sensor pose is within the one or more thresholds.

In some embodiments, maneuvering the robotic vehicle to another locationfrom which to capture another image of the target image may includecomputing or determining a preferred relative pose in response todetermining that the image sensor pose is not within the one or morethresholds, determining a location associated with the preferredrelative pose, and maneuvering the robotic vehicle to the determinedlocation.

In some embodiments, maneuvering the robotic vehicle to another locationfrom which to capture another image of the target image may includemaneuvering the robotic vehicle to one or more other locations,capturing another image of the target image from each of the otherlocations, extracting third features from the another image, matchingone or more extracted third features with one or more extracted secondfeatures, and estimating a second image sensor pose based on matchedsecond and third features and the known dimensions of the target image.

Various embodiments may include capturing an image with the image sensorof the robotic vehicle, extracting first features from the capturedimage, determining whether a number of extracted features exceeds athreshold number of features, maneuvering the robotic vehicle to anotherlocation in response to determining that the number of extractedfeatures exceeds the threshold number of features, obtaining one or moreof a translation of the robotic vehicle and a rotation of the roboticvehicle, determining whether an angular velocity of the robotic vehicleis less than an angular velocity threshold, matching one or moreextracted first features with one or more second features extracted froma captured second image, and estimating an image sensor pose based onmatched features and one of the translation of the robotic vehicle andthe rotation of the robotic vehicle.

In some embodiments, matching one or more extracted first features withone or more second features extracted from a captured second image mayinclude maneuvering the robotic vehicle to another location, capturingand processing a next image from the another location, extracting thesecond features from the next image, and matching the first featuresfrom the first image and the second features from the next image.

Some embodiments may further include estimating a scale between a knownphysical size of an object in the image and an estimated size of theobject based on the image. In some embodiments, the estimated size ofthe object based on the image may be based on matches of the one or moreextracted first features with the one or more second features. Someembodiments may further include determining that the image sensorinitialization is successful in response to determining that the secondimage sensor pose is within the one or more thresholds.

Various embodiments may include capturing a first image with the imagesensor of the robotic vehicle at a first location, extracting firstfeatures from the captured first image, extracting second features froma predefined image pyramid, matching one or more extracted firstfeatures with one or more extracted second features, estimating a firstimage sensor pose based on matched first features and second featuresand known dimensions of the target image, maneuvering the roboticvehicle to a second location and capturing a second image from thesecond location, extracting third features from the captured secondimage, maneuvering the robotic vehicle to a third location and capturinga third image from the third location, extracting fourth features fromthe captured third image, matching one or more extracted third featureswith one or more extracted fourth features, estimating a second imagesensor pose based on the matched third and fourth features, anddetermining a coordinate system for the robotic vehicle based on theimage sensor pose and the second image sensor pose. In some embodiments,the first image may be an image of a target image, and the secondfeatures may be extracted from a predefined image pyramid.

Some embodiments may further include determining that the roboticvehicle has returned to the first location based on the translation ofthe robotic vehicle and the rotation of the robotic vehicle, capturing afourth image with the image sensor of the robotic vehicle at a firstlocation, extracting fifth features from the fourth image, matching oneor more extracted fifth features and with one or more extracted secondfeatures, and estimating a third image sensor pose based on the matchedfifth features and second features, wherein determining the coordinatesystem for the robotic vehicle may include determining the coordinatesystem for the robotic vehicle based on the first image sensor pose, thesecond image sensor pose, and the third image sensor pose.

In some embodiments, determining the coordinate system for the roboticvehicle based on the image sensor pose and the second image sensor posemay include performing a bundle adjustment using the first image sensorpose and the second image sensor pose. In some embodiments, performingthe bundle adjustment using the image sensor pose and the another imagesensor pose may include performing the bundle adjustment using thematched first features and second features. In some embodiments,performing the bundle adjustment using the image sensor pose and theanother image sensor pose may include performing the bundle adjustmentusing the matched third and fourth features. In some embodiments,performing the bundle adjustment using the image sensor pose and theanother image sensor pose may include performing the bundle adjustmentusing the matched fifth features and second features.

Further embodiments may include a robotic vehicle configured withprocessor-executable instructions to perform operations of any of themethods summarized above. Further embodiments may include a processingdevice for use in a robotic vehicle configured to perform operations ofany of the methods summarized above. Further embodiments may include arobotic vehicle including means for performing functions of any of themethods summarized above.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are incorporated herein and constitutepart of this specification, illustrate example embodiments, and togetherwith the general description given above and the detailed descriptiongiven below, serve to explain the features of various embodiments.

FIG. 1 is a system block diagram of a robotic vehicle operating withincommunication system according to various embodiments.

FIG. 2 is a component block diagram illustrating components of a roboticvehicle according to various embodiments.

FIG. 3 is a component block diagram illustrating a processing devicesuitable for use in robotic vehicles implementing various embodiments.

FIG. 4 is a component block diagram illustrating components of an imagecapture and processing system of a robotic vehicle suitable for use withvarious embodiments.

FIG. 5 is a process flow diagram illustrating a method of image sensorinitialization in a robotic vehicle according to various embodiments.

FIG. 6A is a process flow diagram illustrating a method of image sensorinitialization in a robotic vehicle according to various embodiments.

FIG. 6B is a process flow diagram illustrating a method of image sensorinitialization in a robotic vehicle according to various embodiments.

FIG. 6C is a process flow diagram illustrating a method of image sensorinitialization in a robotic vehicle according to various embodiments.

FIG. 7 is a process flow diagram illustrating a method of image sensorinitialization in a robotic vehicle according to various embodiments.

DETAILED DESCRIPTION

Various embodiments will be described in detail with reference to theaccompanying drawings. Wherever possible, the same reference numberswill be used throughout the drawings to refer to the same or like parts.References made to particular examples and embodiments are forillustrative purposes, and are not intended to limit the scope of theclaims.

Various embodiments include methods that may be implemented on aprocessor of a robotic vehicle for initializing an image sensor of arobotic vehicle. In some embodiments, the methods may include methodsfor monocular camera-based SLAM initialization.

As used herein, the term “robotic vehicle” refers to one of varioustypes of vehicles including an onboard processing device configured toprovide some autonomous or semi-autonomous capabilities. Examples ofrobotic vehicles include but are not limited to: aerial vehicles, suchas an unmanned aerial vehicle (UAV); ground vehicles (e.g., anautonomous or semi-autonomous car, a vacuum robot, etc.); water-basedvehicles (i.e., vehicles configured for operation on the surface of thewater or under water); space-based vehicles (e.g., a spacecraft or spaceprobe); and/or some combination thereof. In some embodiments, therobotic vehicle may be manned. In other embodiments, the robotic vehiclemay be unmanned. In embodiments in which the robotic vehicle isautonomous, the robotic vehicle may include an onboard computing deviceconfigured to maneuver and/or navigate the robotic vehicle withoutremote operating instructions (i.e., autonomously), such as from a humanoperator (e.g., via a remote computing device). In embodiments in whichthe robotic vehicle is semi-autonomous, the robotic vehicle may includean onboard computing device configured to receive some information orinstructions, such as from a human operator (e.g., via a remotecomputing device), and autonomously maneuver and/or navigate the roboticvehicle consistent with the received information or instructions. Insome implementations, the robotic vehicle may be an aerial vehicle(unmanned or manned), which may be a rotorcraft or winged aircraft. Forexample, a rotorcraft (also referred to as a multirotor or multicopter)may include a plurality of propulsion units (e.g., rotors/propellers)that provide propulsion and/or lifting forces for the robotic vehicle.Specific non-limiting examples of rotorcraft include tricopters (threerotors), quadcopters (four rotors), hexacopters (six rotors), andoctocopters (eight rotors). However, a rotorcraft may include any numberof rotors. A robotic vehicle may include a variety of components and/orpayloads that may perform a variety of functions.

Robotic vehicles may employ simultaneous localization and mapping (SLAM)techniques to construct and update a map of an unknown environment whilesimultaneously keeping track of the robotic vehicle's location withinthe environment. Robotic vehicles are increasingly equipped with imagesensor devices for capturing images and video. In some embodiments, theimage sensor device may include a monocular image sensor (e.g., amonocular camera). A robotic vehicle may gather data useful for SLAMusing the image sensor device.

For robotic vehicles equipped with a monocular image sensor, the imagesensor may be unable to provide sufficient information to enable theprocessing device to determine both image sensor position andorientation (pose) and map points for the environment. Further, forrobotic vehicles equipped with a monocular image sensor, initializationof the image sensor is important. Errors in initialization of themonocular image sensor may lead to un-correctable bias or errors in therobotic vehicle's coordinate system. Further, the inherent limitationsof a monocular image sensor (i.e., only one point of view, etc.) maypermit initialization errors caused by environmental characteristics,such as illumination, a lack of features in the robotic vehicle'senvironment, and moving objects in the environment.

In various embodiments, a processor device of a robotic vehicle may beconfigured to perform methods of image sensor initialization. In someembodiments, the methods may include methods of monocular camera-basedSLAM initialization. In some embodiments, the processor of the roboticvehicle may capture images of a target image from a plurality oflocations, and may use the multiple images to more accurately estimate apose of an image sensor. For example, poses in a SLAM system may berepresented in a coordinate system that may be based on a reference inthe robotic vehicle's environment. However, pose estimation that reliesonly on images of the target image or only on movement sensors mayinclude errors, and therefore may be inaccurate.

In various embodiments, the processor device of the robotic vehicle mayperform two (or more) estimations, including a first estimate based ontwo or more images captured of a target image (e.g., using an imagesensor, such as a monocular image sensor) and a second estimate usingone or more movement sensors of the robotic vehicle (e.g., a wheelsensor, an inertial measurement sensor, or another similar sensor). Insuch embodiments, the processor may combine the two or more estimations(e.g., via a bundle adjustment or a similar process), and the (processormay generate a more accurate coordinate system based on the result ofthe combined estimations.

In some embodiments, the processor device may extract first featuresfrom a captured image of the target image, and may extract secondfeatures from a predefined image pyramid of the target image. Theprocessor device may match first features and second features, and mayestimate an image sensor (pose based on matched features and knowndimensions of the target image. In response to determining that theimage sensor pose is not within one or more thresholds, the processordevice may maneuver the robotic vehicle to one or more other locationsfrom which to capture another image of the target image. In someembodiments, the processor device of the robotic vehicle may obtain atranslation of the robotic vehicle and/or a rotation of the roboticvehicle, e.g., from one or more other sensors of the robotic vehicle(e.g., a wheel sensor, an inertial-measurement unit, etc.). In suchembodiments, the processor device may estimate the image sensor posebased on matched features and one or more of the translation of therobotic vehicle and the rotation of the robotic vehicle.

Various embodiments may be implemented within a robotic vehicleoperating within a variety of communication systems 100, an example ofwhich is illustrated in FIG. 1. With reference to FIG. 1, thecommunication system 100 may include a robotic vehicle 102, a basestation 104, an access point 106, a communication network 108, and anetwork element 110. In some embodiments, the robotic vehicle 120 may beequipped with an image sensor 102 a. In some embodiments, the imagesensor 102 a may include a monocular image sensor.

The base station 104 and the access point 106 may provide wirelesscommunications to access the communication network 108 over a wiredand/or wireless communication backhaul 116 and 118, respectively. Thebase station 104 may include base stations configured to providewireless communications over a wide area (e.g., macro cells), as well assmall cells, which may include a micro cell, a femto cell, a pico cell,and other similar network access points. The access point 106 mayinclude access points configured to provide wireless communications overa relatively smaller area. Other examples of base stations and accesspoints are also possible.

The robotic vehicle 102 may communicate with the base station 104 over awireless communication link 112, and with the access point 106 over awireless communication link 114. The wireless communication links 112and 114 may include a plurality of carrier signals, frequencies, orfrequency bands, each of which may include a plurality of logicalchannels. The wireless communication links 112 and 114 may utilize oneor more radio access technologies (RATs). Examples of RATs that may beused in a wireless communication link include 3GPP Long Term Evolution(LTE), 3G, 4G, 5G, Global System for Mobility (GSM), Code DivisionMultiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA),Worldwide Interoperability for Microwave Access (WiMAX), Time DivisionMultiple Access (TDMA), and other mobile telephony communicationtechnologies cellular RATs. Further examples of RATs that may be used inone or more of the various wireless communication links within thecommunication system 100 include medium range protocols such as Wi-Fi,LTE-U, LTE-Direct, LAA, MuLTEfire, and relatively short range RATs suchas ZigBee, Bluetooth, and Bluetooth Low Energy (LE).

The network element 110 may include a network server or another similarnetwork element. The network element 110 may communicate with thecommunication network 108 over a communication link 122. The roboticvehicle 102 and the network element 110 may communicate via thecommunication network 108. The network element 110 may provide therobotic vehicle 102 with a variety of information, such as navigationinformation, weather information, information about local air, ground,and/or sea traffic, movement control instructions, and otherinformation, instructions, or commands relevant to operations of therobotic vehicle 102.

In various embodiments, the robotic vehicle 102 may move in anenvironment 120. In some embodiments, the robotic vehicle may use theimage sensor 102 a to capture one or more images of a target image 125in the environment 120. In some embodiments, the target image 125 mayinclude a test image, which may include known characteristics, such as aheight, a width, a texture, and/or other characteristics.

Robotic vehicles may include winged or rotorcraft varieties. FIG. 2illustrates an example robotic vehicle 200 of a ground vehicle designthat utilizes one or more wheels 202 driven by corresponding motors toprovide locomotion to the robotic vehicle 200. The robotic vehicle 200is illustrated as an example of a robotic vehicle that may utilizevarious embodiments, but is not intended to imply or require thatvarious embodiments are limited to ground robotic vehicles. For example,various embodiments may be used with rotorcraft or winged roboticvehicles, water-borne robotic vehicles, and space-based roboticvehicles.

With reference to FIGS. 1 and 2, the robotic vehicle 200 may be similarto the robotic vehicle 102. The robotic vehicle 200 may include a numberof wheels 202, a body 204, and an image sensor 206. The frame 204 mayprovide structural support for the motors and their associated wheels202 as well as for the image sensor 206. For ease of description andillustration, some detailed aspects of the robotic vehicle 200 areomitted such as wiring, frame structure interconnects, or other featuresthat would be known to one of skill in the art. While the illustratedrobotic vehicle 200 has wheels 202, this is merely exemplary and variousembodiments may include any variety of components to provide propulsionand maneuvering capabilities, such as treads, paddles, skids, or anycombination thereof or of other components.

The robotic vehicle 200 may further include a control unit 210 that mayhouse various circuits and devices used to power and control theoperation of the robotic vehicle 200. The control unit 210 may include aprocessor 220, a power module 230, sensors 240, one or more payloadsecuring units 244, one or more image sensors 245, an output module 250,an input module 260, and a radio module 270.

The processor 220 may be configured with processor-executableinstructions to control travel and other operations of the roboticvehicle 200, including operations of various embodiments. The processor220 may include or be coupled to a navigation unit 222, a memory 224, agyro/accelerometer unit 226, and a maneuvering data module 228. Theprocessor 220 and/or the navigation unit 222 may be configured tocommunicate with a server through a wireless connection (e.g., acellular data network) to receive data useful in navigation, providereal-time position reports, and assess data.

The maneuvering data module 228 may be coupled to the processor 220and/or the navigation unit 222, and may be configured to provide travelcontrol-related information such as orientation, attitude, speed,heading, and similar information that the navigation unit 222 may usefor navigation purposes, such as dead reckoning between GlobalNavigation Satellite System (GNSS) position updates. Thegyro/accelerometer unit 226 may include an accelerometer, a gyroscope,an inertial sensor, an inertial measurement unit (IMU), or other similarsensors. The maneuvering data module 228 may include or receive datafrom the gyro/accelerometer unit 226 that provides data regarding theorientation and accelerations of the robotic vehicle 200 that may beused in navigation and positioning calculations, as well as providingdata used in various embodiments for processing images.

The processor 220 may further receive additional information from one ormore image sensors 245 (e.g., a camera, which may be a monocular camera)and/or other sensors 240. In some embodiments, the image sensor(s) 245may include an optical sensor capable of infrared, ultraviolet, and/orother wavelengths of light. The sensors 240 may also include a wheelsensor, a radio frequency (RF) sensor, a barometer, a sonaremitter/detector, a radar emitter/detector, a microphone or anotheracoustic sensor, or another sensor that may provide information usableby the processor 220 for movement operations as well as navigation andpositioning calculations. The sensors 240 may include contact orpressure sensors that may provide a signal that indicates when therobotic vehicle 200 has made contact with a surface. Thepayload-securing units 244 may include an actuator motor that drives agripping and release mechanism and related controls that are responsiveto the control unit 210 to grip and release a payload in response tocommands from the control unit 210.

The power module 230 may include one or more batteries that may providepower to various components, including the processor 220, the sensors240, the payload-securing unit(s) 244, the image sensor(s) 245, theoutput module 250, the input module 260, and the radio module 270. Inaddition, the power module 230 may include energy storage components,such as rechargeable batteries. The processor 220 may be configured withprocessor-executable instructions to control the charging of the powermodule 230 (i.e., the storage of harvested energy), such as by executinga charging control algorithm using a charge control circuit.Alternatively or additionally, the power module 230 may be configured tomanage its own charging. The processor 220 may be coupled to the outputmodule 250, which may output control signals for managing the motorsthat drive the rotors 202 and other components.

The robotic vehicle 200 may be controlled through control of theindividual motors of the rotors 202 as the robotic vehicle 200progresses toward a destination. The processor 220 may receive data fromthe navigation unit 222 and use such data in order to determine thepresent position and orientation of the robotic vehicle 200, as well asthe appropriate course towards the destination or intermediate sites. Invarious embodiments, the navigation unit 222 may include a GNSS receiversystem (e.g., one or more global positioning system (GPS) receivers)enabling the robotic vehicle 200 to navigate using GNSS signals.Alternatively or in addition, the navigation unit 222 may be equippedwith radio navigation receivers for receiving navigation beacons orother signals from radio nodes, such as navigation beacons (e.g., veryhigh frequency (VHF) omni-directional range (VOR) beacons), Wi-Fi accesspoints, cellular network sites, radio station, remote computing devices,other robotic vehicles, etc.

The radio module 270 may be configured to receive navigation signals,such as signals from aviation navigation facilities, etc., and providesuch signals to the processor 220 and/or the navigation unit 222 toassist in robotic vehicle navigation. In various embodiments, thenavigation unit 222 may use signals received from recognizable RFemitters (e.g., AM/FM radio stations, Wi-Fi access points, and cellularnetwork base stations) on the ground.

The radio module 270 may include a modem 274 and a transmit/receiveantenna 272. The radio module 270 may be configured to conduct wirelesscommunications with a variety of wireless communication devices (e.g., awireless communication device (WCD) 290), examples of which include awireless telephony base station or cell tower (e.g., the base station104), a network access point (e.g., the access point 106), a beacon, asmartphone, a tablet, or another computing device with which the roboticvehicle 200 may communicate (such as the network element 110). Theprocessor 220 may establish a bi-directional wireless communication link294 via the modem 274 and the antenna 272 of the radio module 270 andthe wireless communication device 290 via a transmit/receive antenna292. In some embodiments, the radio module 270 may be configured tosupport multiple connections with different wireless communicationdevices using different radio access technologies.

In various embodiments, the wireless communication device 290 may beconnected to a server through intermediate access points. In an example,the wireless communication device 290 may be a server of a roboticvehicle operator, a third party service (e.g., package delivery,billing, etc.), or a site communication access point. The roboticvehicle 200 may communicate with a server through one or moreintermediate communication links, such as a wireless telephony networkthat is coupled to a wide area network (e.g., the Internet) or othercommunication devices. In some embodiments, the robotic vehicle 200 mayinclude and employ other forms of radio communication, such as meshconnections with other robotic vehicles or connections to otherinformation sources (e.g., balloons or other stations for collectingand/or distributing weather or other data harvesting information).

In various embodiments, the control unit 210 may be equipped with aninput module 260, which may be used for a variety of applications. Forexample, the input module 260 may receive images or data from an onboardcamera or sensor, or may receive electronic signals from othercomponents (e.g., a payload).

While various components of the control unit 210 are illustrated in FIG.2 as separate components, some or all of the components (e.g., theprocessor 220, the output module 250, the radio module 270, and otherunits) may be integrated together in a single processing device 310, anexample of which is illustrated in FIG. 3.

With reference to FIGS. 1-3, the processing device 310 may be configuredto be used in a robotic vehicle and may be configured as or including asystem-on-chip (SoC) 312. The SoC 312 may include (but is not limitedto) a processor 314, a memory 316, a communication interface 318, and astorage memory interface 320. The processing device 310 or the SoC 312may further include a communication component 322, such as a wired orwireless modem, a storage memory 324, an antenna 326 for establishing awireless communication link, and/or the like. The processing device 310or the SoC 312 may further include a hardware interface 328 configuredto enable the processor 314 to communicate with and control variouscomponents of a robotic vehicle. The processor 314 may include any of avariety of processing devices, for example any number of processorcores.

The term “system-on-chip” (SoC) is used herein to refer to a set ofinterconnected electronic circuits typically, but not exclusively,including one or more processors (e.g., 314), a memory (e.g., 316), anda communication interface (e.g., 318). The SoC 312 may include a varietyof different types of processors 314 and processor cores, such as ageneral purpose processor, a central processing unit (CPU), a digitalsignal processor (DSP), a graphics processing unit (GPU), an acceleratedprocessing unit (APU), a subsystem processor of specific components ofthe processing device, such as an image processor for a camera subsystemor a display processor for a display, an auxiliary processor, asingle-core processor, and a multicore processor. The SoC 312 mayfurther embody other hardware and hardware combinations, such as a fieldprogrammable gate array (FPGA), an application-specific integratedcircuit (ASIC), other programmable logic device, discrete gate logic,transistor logic, performance monitoring hardware, watchdog hardware,and time references. Integrated circuits may be configured such that thecomponents of the integrated circuit reside on a single piece ofsemiconductor material, such as silicon.

The SoC 312 may include one or more processors 314. The processingdevice 310 may include more than one SoC 312, thereby increasing thenumber of processors 314 and processor cores. The processing device 310may also include processors 314 that are not associated with an SoC 312(i.e., external to the SoC 312). Individual processors 314 may bemulticore processors. The processors 314 may each be configured forspecific purposes that may be the same as or different from otherprocessors 314 of the processing device 310 or SoC 312. One or more ofthe processors 314 and processor cores of the same or differentconfigurations may be grouped together. A group of processors 314 orprocessor cores may be referred to as a multi-processor cluster.

The memory 316 of the SoC 312 may be a volatile or non-volatile memoryconfigured for storing data and processor-executable instructions foraccess by the processor 314. The processing device 310 and/or SoC 312may include one or more memories 316 configured for various purposes.One or more memories 316 may include volatile memories such as randomaccess memory (RAM) or main memory, or cache memory.

Some or all of the components of the processing device 310 and the SoC312 may be arranged differently and/or combined while still serving thefunctions of the various aspects. The processing device 310 and the SoC312 may not be limited to one of each of the components, and multipleinstances of each component may be included in various configurations ofthe processing device 310.

FIG. 4 illustrates an image capture and processing system 400 of arobotic vehicle (e.g., 102, 200 in FIGS. 1 and 2) suitable for use withvarious embodiments. With reference to FIGS. 1-4, the image capture andprocessing system 400 may be implemented in hardware components and/orsoftware components of the robotic vehicle, the operation of which maybe controlled by one or more processors (e.g., the processor 220, theprocessing device 310, the SoC 312, and/or the like) of the roboticvehicle.

An image sensor 406 may capture light of an image 402 that entersthrough a lens 404. The lens 404 may include a fish eye lens or anothersimilar lens that may be configured to provide a wide image captureangle. The image sensor 406 may provide image data to an image signalprocessing (ISP) unit 408. A region of interest (ROI) selection unit 410may provide data to the ISP 408 data for the selection of a region ofinterest within the image data. In some embodiments, the image sensor406 may be similar to the image sensor 102 a, 245.

The ISP 408 may provide image information and ROI selection informationto a rolling-shutter correction, image warp, and crop unit 412. An imagerectification unit 414 may provide information and/or processingfunctions to the rolling-shutter correction, image warp, and crop unit426. In some embodiments, the image rectification unit 414 may provideinformation and/or processing functions to correct for image distortioncaused by the lens 404, an image distortion effect caused by the imagesensor 406 (e.g., distortion such as wobble, skew, smear, and the like),or other image distortion.

The rolling-shutter correction, image warp, and crop unit 412 mayprovide as output a corrected image 416. In some embodiments, thecorrected image 416 may be based on cropping, distortion correction,and/or application of a transformation matrix. In some embodiments, thecorrected image may include an image having a corrected horizontalorientation or horizontal rotation. In some embodiments, the correctedimage may include a stabilized video output. Other examples of acorrected image 416 are possible.

FIG. 5 illustrates a method 500 of image sensor initialization in arobotic vehicle according to various embodiments. With reference toFIGS. 1-5, a processor of a robotic vehicle (e.g., the processor 220,the processing device 310, the SoC 312, and/or the like and hardwarecomponents and/or software components of the robotic vehicle may captureand process an image using an image sensor of the robotic vehicle (e.g.,the image sensor(s) 245).

In block 502, the processor of the robotic vehicle may capture andprocess an image of a target image. For example, using an image sensor(e.g., the image sensor 1102 a, 245) the processor may capture an imagefrom the environment, such as the target image 125. In some embodiments,the processor may process the captured image, for example, the generatedrepresentation such as a pyramid representation of the image. In someembodiments, the processor may generate a low pass pyramid. In someembodiments, the processor may generate a band pass pyramid. In someembodiments, the processor may apply a smoothing kernel for a generatedpyramid, such as a Gaussian pyramid, a Laplacian pyramid, a Steerablepyramid, or another smoothing kernel or similar processing technique.

In block 504, the processor may extract a plurality of features from theprocessed image. For example, the processor may analyze or furtherprocess the captured image, which may include a very large and/or verycomplex set of data, and extract (e.g., identify, select, compute) aplurality features from the processed image. In some embodiments, thefeatures may include attribute information of the image. In someembodiments, the processor may analyze and/or process the captured imageto generate a feature descriptor or a feature vector (e.g., ann-dimensional vector of numerical features that represent the capturedimage). In some embodiments, in addition to the attribute information,the processor may generate additional information such as an edgeorientation, a gradient magnitude, a polarity and/or strength of an edgedetection, relative brightness and/or color information within the image(e.g., blob detection), and other similar additional information.

In block 506, the processor may obtain a predefined image pyramid. Forexample, the processor may obtain from a memory of the robotic vehicle apre-stored or predefined image pyramid. In some embodiments, thepredefined image pyramid may be based on the image in the environment(e.g., the target image 125).

In block 508, the processor of the robotic vehicle may extract aplurality of features from the predefined image pyramid.

In various embodiments, the image in the environment may include testimage, and the processor may be configured to capture an image of thetest image and compare features extracted from the captured image of thetest image and features extracted from a predefined image pyramid of thetest image.

In various embodiments, the operations of blocks 502 and 504 may beperformed before, after, or substantially simultaneously with theoperations of blocks 506 and 508.

In block 510, the processor may match the extracted features of theprocessed image (e.g., from the operations of block 504) with extractedfeatures of the predefined image pyramid (e.g., from the operations ofblock 508). In some embodiments, the processor may identify a set ofpoints in the processed image and a set of points of the predefinedimage pyramid that correspond (e.g., are substantially the same in bothimages) by matching features in the processed image with correspondingfeatures in the predefined image pyramid. In some embodiments, the(processor may determine that two or more points correspond based on ametric such as a degree of error of a descriptor (e.g., within athreshold value or range), a distance between the two or more pointsbeing within a threshold distance, and other similar metrics. In someembodiments, the correspondence may include a degree of translation, adegree of rotation, a difference in illumination, or another difference,distinction, characteristic, or aspect of the two or more points thatmay be compared.

In block 512, the processor may estimate a pose of the image sensor (an“image sensor pose”) based on matched features and known dimensions ofthe image. In some embodiments, the processor may obtain the imagedimensions (e.g., dimensions of the target image 125) from the memory ofthe robotic vehicle. In some embodiments, the processor may performrelative pose estimation using a variety of methods. In someembodiments, the processor may determine an Essential matrix, such as bydetermining a fundamental matrix based on relationship(s) betweenextracted features of the processed image and extracted features of thepredefined image pyramid. In some embodiments, the processor maydetermine a homography matrix based on a relationship (e.g., acorrelation) between features in projections of a plane of the processedimage and the predefined image pyramid. In some embodiments, theprocessor may employ a geometric (e.g., keypoint-based) bundleadjustment to match features of the processed image and of thepredefined image pyramid, such as an Essential matrix based method,homography based method, bundle adjustment, etc.

In block 514, the processor may evaluate the image sensor pose. In someembodiments, the processor may compare the image sensor pose with one ormore thresholds. For example, the processor may compare a relativetranslation of the image sensor pose with a threshold relativetranslation. In some embodiments, the relative translation may representa number of pixels of the captured image. In such embodiments, arelative translation that exceeds the threshold may include a number ofpicture elements (e.g., pixels) that is too few to obtain an imagesensor pose estimation with sufficiently high quality. In someembodiments, the processor may compare the relative rotation of theimage sensor pose (e.g., a relative rotation between the robotic vehicleand the image) with a threshold relative rotation. In some embodiments,the relative rotation may represent a projection angle of the capturedimage. In such embodiments, a relative rotation that exceeds thethreshold may include a projection angle that is too large to obtain animage sensor pose estimation with sufficiently high quality. In someembodiments, the processor may compare a relative illumination of theimage sensor pose with a threshold relative illumination. In someembodiments, the relative elimination may represent an illuminationlevel of one or more pixels, a pixel blob, etc. of the captured image.

In determination block 516, the processor of the robotic vehicle maydetermine whether the image sensor pose is within the one or morethresholds (e.g., the threshold relative translation and/or thethreshold relative rotation).

In response to determining that the image sensor pose is within the oneor more thresholds (i.e., determination block 516=“Yes”), the processormay determine that the image sensor initialization is successful, andmay store the image sensor pose in block 518.

In response to determining that the image sensor pose of the imagesensor is not within the one or more thresholds (i.e., determinationblock 516=“No”), the processor may compute a preferred relative pose toinitialize in block 520. In some embodiments, the preferred relativepose may include a difference between the image (e.g., the target image125) and the image sensor.

In block 522, the processor may determine a location associated with thepreferred relative pose. For example, the robotic vehicle processor maycalculate a location based on a distance between the image sensor poseand the preferred relative pose.

In block 524, the processor maneuver the robotic vehicle to thedetermined location.

The processor may capture and process an image (e.g., of the targetimage 125) in block 502.

FIG. 6A illustrates a method 600 of image sensor initialization in arobotic vehicle according to various embodiments. With reference toFIGS. 1-6A, a processor of a robotic vehicle (e.g., the processor 220,the processing device 310, the SoC 312, and/or the like and hardwarecomponents and/or software components of the robotic vehicle may captureand process an image using an image sensor of the robotic vehicle (e.g.,the image sensor(s) 245).

In block 602, the processor of the robotic vehicle may capture andprocess an image. For example, using an image sensor (e.g., the imagesensor 102 a, 245) the processor may capture an image from theenvironment. In some embodiments, the captured image may be of anyaspect of the environment (e.g., any object or feature). In someembodiments, the captured image may be of a target image, such as thetarget image 125. In some embodiments, the processor may process thecaptured image, for example, the generated representation such as apyramid representation of the image. In some embodiments, the operationsof block 602 may be similar to the operations of block 502 of the method500.

In block 604, the processor may extract a plurality of features from theprocessed image. For example, the processor may analyze or furtherprocess the captured image, which may include a very large and/or verycomplex set of data, and extract (e.g., identify, select, compute) aplurality features from the processed image. In some embodiments, theoperations of block 604 may be similar to the operations of block 504 ofthe method 500.

In determination block 606, the processor of the robotic vehicle maydetermine whether a number of extracted features is greater than athreshold number of features. For example, the processor may determinewhether the number of features extracted is sufficiently large (i.e.,exceeds a threshold number of features) for use in determining a pose ofthe image sensor.

In response to determining that the number of features extracted is notgreater than the threshold number of features (i.e., determination block606=“No”), the processor may again capture and process an image in block602.

In response to determining that the number of features extracted isgreater than the threshold number of features (i.e., determination block606=“Yes”), the processor may maneuver the robotic vehicle to anotherlocation in block 608.

In block 610, the processor may obtain a translation and/or a rotationof the robotic vehicle from sensors of the robotic vehicle. For example,the processor may obtain from an inertial measurement unit, a wheelencoder, accelerometer, or another sensor information that the processormay use to determine the translation and/or the rotation of the roboticvehicle.

In determination block 612, the processor may determine whether anangular velocity of the robotic vehicle is less than an angular velocitythreshold (e.g., AV TH).

In response to determining that the angular velocity is not less thanthe angular velocity threshold (i.e., determination block 612=“No”), theprocessor may again capture and process an image in block 602.

In response to determining that the angular velocity is less than theangular velocity threshold (i.e., determination block 612=“Yes”), theprocessor may set an initial frame, and may store the obtainedtranslation and/or rotation in block 614.

In block 616, the processor may maneuver the robotic vehicle to anotherlocation.

In block 618, the processor may capture and process a next image (e.g.,another image, such as an image from the environment. For example, at alater time, the processor may capture and process a second (orsubsequent) image (which may be of the target image 125).

In block 620, the processor may extract features from the processedimage.

In determination block 622, the processor of the robotic vehicle maydetermine whether a number of features extracted from the next image isgreater than a threshold number of features. For example, the processormay determine whether the number of features extracted is sufficientlylarge (i.e., exceeds a threshold number of features) for use indetermining a pose of the image sensor.

In response to determining that the number of features extracted is notgreater than the threshold number of features (i.e., determination block622=“No”), the processor may continue to capture and process images ofthe target image in block 618.

In response to determining that the number of features extracted isgreater than the threshold number of features i.e., determination block622=“Yes”), the processor may match the extracted features of theprocessed image (e.g., from the operations of block 604) with extractedfeatures of the second (or subsequent) image (e.g., from the operationsof block 620) in block 624.

In block 626, the processor may obtain the translation and/or therotation of the robotic vehicle from movement sensors of the roboticvehicle. In some embodiments, the obtained translation and/or rotationof the robotic vehicle may be related to the capture and processing ofthe second image of the target image (e.g., from the operations of block618).

In block 628, the processor may estimate the pose of the image sensor(the image sensor pose) based on matched features and the translationand/or rotation of the robotic vehicle. In some embodiments, theprocessor may estimate the pose of the image sensor based on acombination of matched features from two or more images (e.g., theinitial frame and one or more next images). In some embodiments, theprocessor may estimate the pose of the image sensor based on acombination of translations and/or rotations associated with the two ormore images.

In determination block 630, the processor of the robotic vehicle maydetermine whether the image sensor pose is within the one or morethresholds (e.g., the threshold relative translation and/or thethreshold relative rotation).

In response to determining that the image sensor pose is not within theone or more thresholds (i.e., determination block 630=“No”), theprocessor may maneuver the robotic vehicle to another location in block616, and may capture and process a next image of the target image inblock 618.

In response to determining that the image sensor pose is within the oneor more thresholds (i.e., determination block 630=“Yes”), the processormay determine that the image sensor initialization is successful, andmay store the image sensor pose in block 632.

FIG. 6B illustrates a method 650 that may be performed in block 630(FIG. 6A). With reference to FIGS. 1-6B, the operations of the method650 may be performed by a processor of a robotic vehicle (e.g., theprocessor 220, the processing device 310, the SoC 312, and/or the likeand hardware components and/or software components of the roboticvehicle).

In block 652, the processor may obtain rotation and translationinformation from the image sensor for the initial frame (e.g., the firstimage of the target image).

In block 654, the processor may obtain rotation and translationinformation from the image sensor for the next image of the targetimage.

In block 656, the processor may compute a relative rotation matrix(e.g., Ra) and a translation matrix (e.g., Ta) of the image information(e.g., based on a comparison of, or differences between, the rotationand translation information for the initial frame and for the nextimage). In various embodiments, the relative rotation and translation ofthe image information may be in the form of matrices or vectors.

In block 658, the processor may obtain rotation and translationinformation of the robotic vehicle from one or more movement sensors forthe initial frame.

In block 660, the processor may obtain rotation and translationinformation of the robotic vehicle from one or more movement sensors forthe next image.

In block 662, the processor may compute a relative rotation matrix(e.g., Rb) and translation matrix (e.g., Tb) of the robotic vehicle(e.g., based on a comparison of, or differences between, the rotationand translation information from the one or more movement sensors forthe initial frame and for the next image).

In determination block 664, the processor may determine whether thecomputed relative rotation matrix of the image information (e.g., Ra) isapproximately equal to the computed relative rotation matrix of therobotic vehicle (e.g., Rb). For example, the processor may determinewhether the computed relative rotation matrix or vector of the imageinformation (e.g., Ra) differs from the computed relative rotationmatrix or vector of the robotic vehicle (e.g., Rb) by a similarity(third) threshold.

In response to determining that the relative rotation of the imageinformation is not approximately equal to the relative rotation of therobotic vehicle (e.g., Ra is not ≈Rb) (i.e., determination block664=“No”), the processor may again obtain rotation and translationinformation from the image sensor for the initial frame in block 652 andobtain rotation and translation information from one or more movementsensors for the initial frame in block 658.

In response to determining that the relative rotation of the imageinformation is approximately equal to the relative rotation of therobotic vehicle (e.g., Ra≈Rb) (i.e., determination block 664=“Yes”), theprocessor may determine that the image information translation matrix(e.g., Ta) is equal to the translation matrix of the robotic vehicle(e.g., Tb), and may calculate a scalar value (e.g., Scale A) for theimage pose equal to a determinant of the translation matrix of therobotic vehicle (e.g., |Tb|) over a determinant of the translationmatrix of the image information (e.g., |Ta|) in block 666. In someembodiments, the scalar value for the image pose (e.g., Scale A) mayinclude a scale between the estimated image size and a known physicalsize of the target image.

In some embodiments, the processor may improve the accuracy of thecomputation of the scalar value for the image pose by obtaininginformation from a plurality of next images. For example, the processormay maneuver the robotic vehicle to a plurality of locations. Theprocessor may capture and process an image of the target image from oneor more of the plurality of locations. The processor may extractfeatures from such images. The processor may also obtain rotation and/ortranslation information from one or more other sensors for each of theplurality of locations. The processor may use a combination of therotation and translation information for each of the images obtainedfrom the plurality of locations, as well as the rotation and translationfor the initial frame, to compute a scalar value for the image posebased on the rotation and translation information of the images of thetarget image and the rotation and translation information from the oneor more sensors from the plurality of locations.

For example, FIG. 6C illustrates a method 670 that may be performed toimprove the accuracy of image pose estimation. With reference to FIGS.1-6C, the operations of the method 670 may be performed by a processorof a robotic vehicle (e.g., the processor 220, the processing device310, the SoC 312, and/or the like and hardware components and/orsoftware components of the robotic vehicle). In blocks 652-666, thedevice processor may perform operations of like-numbered blocks of themethod 650.

In block 672, the processor may maneuver the robotic vehicle to anotherlocation.

In block 674, the processor may capture and process another image. Insome embodiments, the processor may capture and process another image ofthe target image 125. For example, at a later time, the processor maycapture and process a second (or subsequent) image. In some embodiments,the captured image may be of any aspect of the environment (e.g., theobject or feature of which the image is captured in operation 602). Insome embodiments, the captured image may be of a target image, such asthe target image 125.

In block 676, the processor may extract features from the processedimage.

In block 678, the processor may match the extracted features of theprocessed image with extracted features of the next image.

In block 680, the processor may estimate a scale (e.g., Scale C) betweenan estimated image size of the target image determined from the nextimage and the known physical size of the target image.

In block 682, the processor may determine a scale based on a combinationof two or more estimated scales (e.g., Scale A, Scale C). In someembodiments, the processor may use the determined scale to estimate theimage sensor pose.

For example, for both image sensor and movement sensors, the processormay determine a relative rotation from the initial frame to a next frameusing information from the image sensor, and information from one ormore other sensors. In some embodiments, using such information theprocessor may compute a relative rotation, and may further compute atranslation and scale corresponding to the image sensor. In variousembodiments, the processor may assume that a relative pose and samplingtime between the image sensor and the one or more other sensors havebeen calibrated such that they may be ignored in various computationsperformed by the processor.

FIG. 7 illustrates a method 700 of image sensor initialization in arobotic vehicle according to various embodiments. With reference toFIGS. 1-7, a processor of a robotic vehicle (e.g., the processor 220,the processing device 310, the SoC 312, and/or the like) and hardwarecomponents and/or software components of the robotic vehicle may captureand process an image using an image sensor of the robotic vehicle (e.g.,the image sensor(s) 245). In blocks 502-512, the processor may performoperations of like-numbered blocks of the method 500.

In block 702, the processor may maneuver the robotic vehicle. In someembodiments, the processor may maneuver the robotic vehicle to alocation that is different from a first location (e.g., the locationfrom which the image of the target image is captured in block 502, or alocation where the operations of blocks 502-512 are performed). In someembodiments, the processor may maneuver the robotic vehicle according toa pattern, such as a loop, a geometric shape, or another pattern.

In block 704, the processor may capture and process another image.

In block 706, the processor may extract a plurality of features from theprocessed image. For example, the processor may analyze or furtherprocess the captured image, which may include a very large and/or verycomplex set of data, and extract (e.g., identify, select, compute) aplurality features from the processed image.

In various embodiments, the processor may repeat the operations ofblocks 702-706 at least once. For example, the processor may maneuverthe robotic vehicle to a first position, capture and process anotherimage from the first position, and extract features from the processedimage, and then the processor may maneuver the robotic vehicle to asecond session capture and process another image from the secondposition, and extract features from the processed image from the secondposition.

In block 708, the processor may match the extracted features of theprocessed image (e.g., from the operations of block 706) with extractedfeatures of an image pyramid. For example, the processor may matchfeatures extracted from the processed image that was captured at thesecond position with extracted features of processed image that wascaptured at the first position (e.g., an image pyramid).

In block 710, the processor may estimate a second pose of the imagesensor (a “second image sensor pose”) for each processed image based onthe matched features. In determination block 712, the processor maydetermine whether the robotic vehicle has returned to its startinglocation.

In response to determining that the robotic vehicle has not returned toits starting location (e.g., determination block 712=“No”), theprocessor may maneuver the robotic vehicle to another location in block702.

In response to determining that the robotic vehicle has returned to itsstarting location (e.g., determination block 712=“Yes”), the processormay capture and process another image of the target image

In block 716, the processor may extract a plurality of features from theprocessed image.

In block 718, the processor may match the extracted features of theprocessed image with extracted features of the predefined image pyramid.

In block 720, the processor may estimate a third pose of the imagesensor (a “third image sensor pose”) based on matched features anddimensions of the image. In some embodiments, the processor may obtainthe image dimensions (e.g., dimensions of the test image) from thememory of the robotic vehicle. In some embodiments, processor maydetermine the third image sensor pose based on the images captured, thefeatures extracted, the first image sensor pose, and/or the second imagesensor pose.

In block 722, the processor may evaluate the third image sensor pose.

In determination block 724, the processor of the robotic vehicle maydetermine whether the third image sensor pose is within the one or morethresholds (e.g., a threshold relative translation and/or a thresholdrelative rotation).

In response to determining that the third image sensor pose is notwithin the one or more thresholds (i.e., determination block 724=“No”),the processor may capture and process another image of the target imagein block 714.

In response to determining that the third image sensor pose is withinthe one or more thresholds (i.e., determination block 724=“Yes”), theprocessor may perform a bundle adjustment on a plurality of poses and/ormatched features in block 726. In some embodiments, the processor mayperform the bundle adjustment using the first image sensor pose and thesecond image sensor pose(s). In some embodiments, the processor mayperform the bundle adjustment using the first image sensor pose, thesecond image sensor pose(s), and the third image sensor pose. In someembodiments, the processor may perform the bundle adjustment using theextracted features of the processed image (e.g., from the operations ofblock 706) that are matched with the extracted features of thepredefined image pyramid (e.g., from the operations of block 508). Insome embodiments, the processor may perform the bundle adjustment usingthe extracted features of the another image (e.g., from the operationsof block 716) that are matched with the extracted features of thepredefined image pyramid.

In block 728, the processor may determine a coordinate system for therobotic vehicle. In some embodiments, the processor may determine thecoordinate system based on the bundle adjustment using the first imagesensor pose, the second image sensor pose(s), and/or the third imagesensor pose.

In block 730, the processor may store the poses and the extractedfeatures (e.g., as described with respect to the operations of blocks504, 512, 706, 710, 716, and 722).

Various embodiments enable the processor of the robotic vehicle toimprove the initialization of an image sensor of the robotic vehicle.Various embodiments also improve the accuracy of the robotic vehicle'sSLAM capabilities using a more accurately initialize image sensor.Various embodiments also improve capability of a robotic vehicle toinitialize a monocular image sensor for use with SLAM determinations.

Various embodiments illustrated and described are provided merely asexamples to illustrate various features of the claims. However, featuresshown and described with respect to any given embodiment are notnecessarily limited to the associated embodiment and may be used orcombined with other embodiments that are shown and described. Further,the claims are not intended to be limited by any one example embodiment.For example, one or more of the operations of the methods 500, 600, 650,670, and 700 may be substituted for or combined with one or moreoperations of the methods 500, 600, 650, 670, and 700, and vice versa.

The foregoing method descriptions and the process flow diagrams areprovided merely as illustrative examples and are not intended to requireor imply that the operations of various embodiments must be performed inthe order presented. As will be appreciated by one of skill in the artthe order of operations in the foregoing embodiments may be performed inany order. Words such as “thereafter,” “then,” “next,” etc. are notintended to limit the order of the operations; these words are used toguide the reader through the description of the methods. Further, anyreference to claim elements in the singular, for example, using thearticles “a,” “an,” or “the” is not to be construed as limiting theelement to the singular.

Various illustrative logical blocks, modules, circuits, and algorithmoperations described in connection with the embodiments disclosed hereinmay be implemented as electronic hardware, computer software, orcombinations of both. To clearly illustrate this interchangeability ofhardware and software, various illustrative components, blocks, modules,circuits, and operations have been described above generally in terms oftheir functionality. Whether such functionality is implemented ashardware or software depends upon the particular application and designconstraints imposed on the overall system. Skilled artisans mayimplement the described functionality in varying ways for eachparticular application, but such embodiment decisions should not beinterpreted as causing a departure from the scope of the claims.

The hardware used to implement various illustrative logics, logicalblocks, modules, and circuits described in connection with the aspectsdisclosed herein may be implemented or performed with a general purposeprocessor, a digital signal processor (DSP), an application specificintegrated circuit (AMC), a field programmable gate array (FPGA) orother programmable logic device, discrete gate or transistor logic,discrete hardware components, or any combination thereof designed toperform the functions described herein. A general-purpose processor maybe a microprocessor, but, in the alternative, the processor may be anyconventional processor, controller, microcontroller, or state machine. Aprocessor may also be implemented as a combination of receiver smartobjects, e.g., a combination of a DSP and a microprocessor, a pluralityof microprocessors, one or more microprocessors in conjunction with aDSP core, or any other such configuration. Alternatively, someoperations or methods may be performed by circuitry that is specific toa given function.

In one or more aspects, the functions described may be implemented inhardware, software, firmware, or any combination thereof. If implementedin software, the functions may be stored as one or more instructions orcode on a non-transitory computer-readable storage medium ornon-transitory processor-readable storage medium. The operations of amethod or algorithm disclosed herein may be embodied in aprocessor-executable software module or processor-executableinstructions, which may reside on a non-transitory computer-readable orprocessor-readable storage medium. Non-transitory computer-readable orprocessor-readable storage media may be any storage media that may beaccessed by a computer or a processor. By way of example but notlimitation, such non-transitory computer-readable or processor-readablestorage media may include RAM, ROM, EEPROM, FLASH memory, CD-ROM orother optical disk storage, magnetic disk storage or other magneticstorage smart objects, or any other medium that may be used to storedesired program code in the form of instructions or data structures andthat may be accessed by a computer. Disk and disc, as used herein,includes compact disc (CD), laser disc, optical disc, digital versatiledisc (DVD), floppy disk, and Blu-ray disc where disks usually reproducedata magnetically, while discs reproduce data optically with lasers.Combinations of the above are also included within the scope ofnon-transitory computer-readable and processor-readable media.Additionally, the operations of a method or algorithm may reside as oneor any combination or set of codes and/or instructions on anon-transitory processor-readable storage medium and/orcomputer-readable storage medium, which may be incorporated into acomputer program product.

The preceding description of the disclosed embodiments is provided toenable any person skilled in the art to make or use the claims. Variousmodifications to these embodiments will be readily apparent to thoseskilled in the art, and the generic principles defined herein may beapplied to other embodiments without departing from the spirit or scopeof the claims. Thus, the present disclosure is not intended to belimited to the embodiments shown herein but is to be accorded the widestscope consistent with the following claims and the principles and novelfeatures disclosed herein.

What is claimed is:
 1. A method of image sensor initialization of animage sensor of a robotic vehicle, comprising: capturing an image of atarget image with the image sensor of the robotic vehicle; extracting,by a processor, first features from the captured image; extracting, bythe processor, second features from a predefined image pyramid of thetarget image; matching, by the processor, one or more extracted firstfeatures with one or more extracted second features; estimating, by theprocessor, an image sensor pose based on matched features and knowndimensions of the target image; determining, by the processor, whetherthe image sensor pose is within one or more thresholds; and in responseto determining that the image sensor pose is not within the one or morethresholds: computing, by the processor, a preferred relative imagesensor pose in response to determining that the image sensor pose is notwithin the one or more thresholds, wherein the preferred relative imagesensor pose comprises a difference between the image and the imagesensor; determining, by the processor, a location associated with thepreferred relative pose; and maneuvering the robotic vehicle to thedetermined location from which to capture another image of the targetimage.
 2. The method of claim 1, wherein the image sensor initializationcomprises monocular camera-based simultaneous localization and mapping(SLAM) initialization.
 3. The method of claim 1, further comprising:determining, by the processor, that the image sensor initialization issuccessful in response to determining that the image sensor pose iswithin the one or more thresholds.
 4. The method of claim 1, whereinmaneuvering the robotic vehicle to another location from which tocapture another image of the target image comprises: maneuvering therobotic vehicle to one or more other locations; capturing, by theprocessor, another image of the target image from each of the otherlocations; extracting, by the processor, third features from the anotherimage; matching, by the processor, one or more extracted third featureswith one or more extracted second features; and estimating, by theprocessor, a second image sensor pose based on matched second and thirdfeatures and the known dimensions of the target image.
 5. A method ofimage sensor initialization of an image sensor of a robotic vehicle,comprising: capturing an image with the image sensor of the roboticvehicle; extracting, by a processor, first features from the capturedimage; determining, by the processor, whether a number of extractedfeatures exceeds a threshold number of features; maneuvering the roboticvehicle to another location in response to determining that the numberof extracted features exceeds the threshold number of features;obtaining, by the processor, one or more of a translation of the roboticvehicle and a rotation of the robotic vehicle using a movement sensor ofthe robotic vehicle; determining, by the processor, whether an angularvelocity of the robotic vehicle is less than an angular velocitythreshold; matching, by the processor, one or more extracted firstfeatures with one or more second features extracted from a capturedsecond image; estimating, by the processor, an image sensor pose basedon matched features and one of the translation of the robotic vehicleand the rotation of the robotic vehicle; computing a first relativerotation matrix based on image information from the captured image and asecond relative rotation matrix based on the one or more of thetranslation of the robotic vehicle and the rotation of the roboticvehicle obtained using the movement sensors of the robotic vehicle; anddetermining whether the first relative rotation matrix is approximatelyequal to the second relative rotation matrix.
 6. The method of claim 5,wherein matching, by the processor, one or more extracted first featureswith one or more second features extracted from a captured second imagecomprises: maneuvering the robotic vehicle to another location;capturing and processing, by the processor, a next image from theanother location; extracting, by the processor, the second features fromthe next image; and matching the first features from the first image andthe second features from the next image.
 7. The method of claim 5,further comprising: estimating a scale between a known physical size ofan object in the image and an estimated size of the object based on theimage.
 8. The method of claim 7, wherein the estimated size of theobject based on the image is based on matches of the one or moreextracted first features with the one or more second features.
 9. Themethod of claim 5, further comprising: determining, by the processor,that the image sensor initialization is successful in response todetermining that the second image sensor pose is within the one or morethresholds.
 10. A method of image sensor initialization of an imagesensor of a robotic vehicle, comprising: capturing a first image withthe image sensor of the robotic vehicle at a first location; extracting,by a processor, first features from the captured first image;extracting, by the processor, second features from a predefined imagepyramid; matching, by the processor, one or more extracted firstfeatures with one or more extracted second features; estimating, by theprocessor, a first image sensor pose based on matched first features andsecond features and known dimensions of the target image; maneuveringthe robotic vehicle to a second location and capturing a second imagefrom the second location; extracting, by the processor, third featuresfrom the captured second image; maneuvering the robotic vehicle to athird location and capturing a third image from the third location;extracting, by the processor, fourth features from the captured thirdimage; matching, by the processor, one or more extracted third featureswith one or more extracted fourth features; estimating, by theprocessor, a second image sensor pose based on the matched third andfourth features; determining, by the processor, that the robotic vehiclehas returned to the first location based on a translation of the roboticvehicle and a rotation of the robotic vehicle; capturing a fourth imagewith the image sensor of the robotic vehicle at the first location inresponse to determining that the robotic vehicle has returned to thefirst location; extracting, by the processor, fifth features from thefourth image; matching, by the processor, one or more extracted fifthfeatures and with one or more extracted second features; estimating, bythe processor, a third image sensor pose based on the matched fifthfeatures and second features; and determining the coordinate system forthe robotic vehicle based on the first image sensor pose, the secondimage sensor pose, and the third image sensor pose.
 11. The method ofclaim 10, wherein the first image is an image of a target image, andwherein the second features are extracted from a predefined imagepyramid.
 12. The method of claim 10, wherein determining the coordinatesystem for the robotic vehicle based on the image sensor pose and thesecond image sensor pose comprises: performing a bundle adjustment usingthe first image sensor pose and the second image sensor pose.
 13. Themethod of claim 12, wherein performing the bundle adjustment using theimage sensor pose and the another image sensor pose comprises:performing the bundle adjustment using the matched first features andsecond features.
 14. The method of claim 12, wherein performing thebundle adjustment using the image sensor pose and the another imagesensor pose comprises: performing the bundle adjustment using thematched third and fourth features.
 15. The method of claim 12, whereinperforming the bundle adjustment using the image sensor pose and theanother image sensor pose comprises: performing the bundle adjustmentusing the matched fifth features and second features.
 16. A roboticvehicle, comprising: an image sensor; a processor coupled to the imagesensor and configured with processor-executable instructions to: capturean image of a target image with the image sensor; extract first featuresfrom the captured image; extract second features from a predefined imagepyramid of the target image; match one or more extracted first featureswith one or more extracted second features; estimate an image sensorpose based on matched features and known dimensions of the target image;determine whether the image sensor pose is within one or morethresholds; and in response to determining that the image sensor pose isnot within the one or more thresholds: compute a preferred relativeimage sensor pose in response to determining that the image sensor poseis not within the one or more thresholds; determine a locationassociated with the preferred relative pose; and maneuver the roboticvehicle to the determined location from which to capture another imageof the target image.
 17. The robotic vehicle of claim 16, wherein theprocessor is further configured with processor-executable instructionssuch that the image sensor initialization comprises a monocularcamera-based simultaneous localization and mapping (SLAM)initialization.
 18. The robotic vehicle of claim 16, wherein theprocessor is further configured with processor-executable instructionsto: determine that the image sensor initialization is successful inresponse to determining that the image sensor pose is within the one ormore thresholds.
 19. The robotic vehicle of claim 16, wherein theprocessor is further configured with processor-executable instructionsto: maneuver the robotic vehicle to one or more other locations; captureanother image of the target image from each of the other locations;extract third features from the another image; match one or moreextracted third features with one or more extracted second features; andestimate a second image sensor pose based on matched second and thirdfeatures and the known dimensions of the target image.
 20. A roboticvehicle, comprising: an image sensor; a processor coupled to the imagesensor and configured with processor-executable instructions to: capturea first image with the image sensor; extract first features from thecaptured image; determine whether a number of extracted features exceedsa threshold number of features; maneuver the robotic vehicle to anotherlocation in response to determining that the number of extractedfeatures exceeds the threshold number of features; obtain one or more ofa translation of the robotic vehicle and a rotation of the roboticvehicle using a movement sensor of the robotic vehicle; determinewhether an angular velocity of the robotic vehicle is less than anangular velocity threshold; match one or more extracted first featureswith one or more second features extracted from a captured second image;estimate an image sensor pose based on matched features and one of thetranslation of the robotic vehicle and the rotation of the roboticvehicle; compute a first relative rotation matrix based on imageinformation from the captured image and a second relative rotationmatrix based on the one or more of the translation of the roboticvehicle and the rotation of the robotic vehicle obtained using themovement sensors of the robotic vehicle; and determine whether the firstrelative rotation matrix is approximately equal to the second relativerotation matrix.
 21. The robotic vehicle of claim 20, wherein theprocessor is further configured with processor-executable instructionsto: maneuver the robotic vehicle to another location; capture andprocess a next image from the another location; extract the secondfeatures from the next image; and match the first features from thefirst image and the second features from the next image.
 22. The roboticvehicle of claim 20, wherein the processor is further configured withprocessor-executable instructions to: estimate a scale between a knownphysical size of an object in the first image and an estimated size ofthe object based on the first image.
 23. The robotic vehicle of claim22, wherein the processor is further configured withprocessor-executable instructions such that the estimated size of theobject based on the first image is based on matches of the one or moreextracted first features with the one or more second features.
 24. Therobotic vehicle of claim 20, wherein the processor is further configuredwith processor-executable instructions to: determine that the imagesensor initialization is successful in response to determining that thesecond image sensor pose is within the one or more thresholds.
 25. Arobotic vehicle, comprising: an image sensor; a processor coupled to theimage sensor and configured with processor-executable instructions to:capture a first image with the image sensor at a first location; extractfirst features from the captured first image; extract second featuresfrom a predefined image pyramid; match one or more extracted firstfeatures with one or more extracted second features; estimate a firstimage sensor pose based on matched first features and second featuresand known dimensions of the target image; maneuver the robotic vehicleto a second location and capturing a second image from the secondlocation; extract third features from the captured second image;maneuver the robotic vehicle to a third location and capturing a thirdimage from the third location; extract fourth features from the capturedthird image; match one or more extracted third features with one or moreextracted fourth features; estimate a second image sensor pose based onthe matched third and fourth features; determine that the roboticvehicle has returned to the first location based on a translation of therobotic vehicle and a rotation of the robotic vehicle; capture a fourthimage with the image sensor of the robotic vehicle at the first locationin response to determining that the robotic vehicle has returned to thefirst location; extract fifth features from the fourth image; match oneor more extracted fifth features and with one or more extracted secondfeatures; estimate a third image sensor pose based on the matched fifthfeatures and second features; and determine the coordinate system forthe robotic vehicle based on the first image sensor pose, the secondimage sensor pose, and the third image sensor pose.
 26. The roboticvehicle of claim 25, wherein the first image is an image of a targetimage, and wherein the second features are extracted from a predefinedimage pyramid.
 27. The robotic vehicle of claim 25, wherein theprocessor is further configured with processor-executable instructionsto: perform a bundle adjustment using the first image sensor pose andthe second image sensor pose.
 28. The robotic vehicle of claim 27,wherein the processor is further configured with processor-executableinstructions to: perform the bundle adjustment using the matched firstfeatures and second features.
 29. The robotic vehicle of claim 27,wherein the processor is further configured with processor-executableinstructions to: perform the bundle adjustment using the matched thirdand fourth features.
 30. The robotic vehicle of claim 27, wherein theprocessor is further configured with processor-executable instructionsto: perform the bundle adjustment using the matched fifth features andsecond features.